Add `Dataset.drop_dims` #2767

kmsquire · 2019-02-13T04:15:04Z

Closes Removing dimensions from Dataset objects #1949
Tests added
Fully documented, including whats-new.rst for all changes and api.rst for new API

pep8speaks · 2019-02-13T04:15:14Z

Hello @kmsquire! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2019-03-03 02:27:43 UTC

kmsquire · 2019-02-13T06:38:43Z

I was looking for this functionality and ran across #1949.

There was one bit of behavior that was unclear to me. Say I have

In [6]: data = xr.Dataset({'A': (['x', 'y'], np.random.randn(2, 3)),
   ...:                    'B': ('x', np.random.randn(2)),
   ...:                    'x': ['a', 'b'], 'z': np.pi})

In [7]: data
Out[7]:
<xarray.Dataset>
Dimensions:  (x: 2, y: 3)
Coordinates:
  * x        (x) <U1 'a' 'b'
Dimensions without coordinates: y
Data variables:
    A        (x, y) float64 -0.662 -0.705 -0.1866 -0.8655 0.5816 -0.2308
    B        (x) float64 -0.1002 2.402
    z        float64 3.142

In [8]: data.drop_dims('x')
Out[8]:
<xarray.Dataset>
Dimensions:  ()
Data variables:
    z        float64 3.142

In this case, I dropped x, but because that eliminated variable A, I also dropped my y dimension, since A was the only variable using y.

At one point, I implemented this slightly differently, which instead gave

In [8]: data.drop_dims('x')
Out[8]:
<xarray.Dataset>
Dimensions:  (y: 3)
Dimensions without coordinates: y
Data variables:
    z        float64 3.142

In some ways, this seems preferable, especially if y has coordinates. The only downside I can see is that this version called Dataset._replace_vars_and_dims(), the docstring for which says it's deprecated (although it's still widely used).

kmsquire · 2019-02-21T17:11:05Z

@shoyer I'm sure you're busy... wondering if you might have time to review this (or direct it to someone else to review)?
Other than my question above, it should be pretty straightforward.

shoyer · 2019-02-22T01:41:41Z

It's OK to use _replace_with_new_dims still. I think we'll eventually replace it in favor of _replace_with_new_dims but we aren't there yet.

I agree that it makes sense to recompute dimensions from variables. There's not much point in keeping around extra dimensions that don't have corresponding variables, and I'm pretty sure that this is consistent with how Dataset.drop() works.

kmsquire · 2019-02-22T16:47:11Z

I agree that it makes sense to recompute dimensions from variables.

Okay, that's what this PR already did (via the call to _replace_with_new_dims).

Rebased on master. From my perspective, this should be good to go.

xarray/core/dataset.py

doc/indexing.rst

kmsquire · 2019-02-26T19:02:23Z

@shoyer I force pushed the requested changes. Please let me know if there is anything else. Thanks!

shoyer · 2019-02-27T00:24:21Z

xarray/core/dataset.py

+        coord_names = set(k for k in self._coord_names if k in variables)
+
+        return self._replace_with_new_dims(
+            variables, coord_names, inplace=inplace)


inplace is now an undefined variable, so your tests are failing

Whoops! I took that out of the tests, but forgot to take it out of dataset.py. Should be fixed now.

* Drops full dimensions and any corresponding variables in a Dataset * Fixes GH1949

kmsquire · 2019-03-03T02:28:02Z

Rebased to fix the NEWS conflict.

shoyer · 2019-03-03T19:43:11Z

thanks @kmsquire, especially for your patience here!

kmsquire · 2019-03-04T01:22:04Z

No worries--thanks for merging!

* upstream/master: Rework whats-new for 0.12 Add whats-new for 0.12.1 Release 0.12.0 enable loading remote hdf5 files (pydata#2782) Push back finalizing deprecations for 0.12 (pydata#2809) Drop failing tests writing multi-dimensional arrays as attributes (pydata#2810) some docs updates (pydata#2746) Add support for cftime.datetime coordinates with coarsen (pydata#2778) Don't use deprecated np.asscalar() (pydata#2800) Improve name concat (pydata#2792) Add `Dataset.drop_dims` (pydata#2767) Quarter offset implemented (base is now latest pydata-master). (pydata#2721) Add use_cftime option to open_dataset (pydata#2759) Bugfix/reduce no axis (pydata#2769) 'standard' now refers to 'gregorian' in cftime_range (pydata#2771)

* ENH: Add Dataset.drop_dims() * Drops full dimensions and any corresponding variables in a Dataset * Fixes GH1949 * DOC: Add Dataset.drop_dims() documentation

…ns with size>1 (#2757) * Quarter offset implemented (base is now latest pydata-master). (#2721) * Quarter offset implemented (base is now latest pydata-master). * Fixed issues raised in review (#2721 (review)) * Updated whats-new.rst with info on quarter offset support. * Updated whats-new.rst with info on quarter offset support. * Update doc/whats-new.rst Co-Authored-By: jwenfai <jwenfai@gmail.com> * Added support for quarter frequencies when resampling CFTimeIndex. Less redundancy in CFTimeIndex resampling tests. * Removed normalization code (unnecessary for cftime_range) in cftime_offsets.py. Removed redundant lines in whats-new.rst. * Removed invalid option from _get_day_of_month docstring. Added tests back in that raises ValueError when resampling (base=24 when resampling to daily freq, e.g., '8D'). * Minor edits to docstrings/comments * lint * Add `Dataset.drop_dims` (#2767) * ENH: Add Dataset.drop_dims() * Drops full dimensions and any corresponding variables in a Dataset * Fixes GH1949 * DOC: Add Dataset.drop_dims() documentation * Improve name concat (#2792) * Added tests of desired name inferring behaviour * Infers names * updated what's new * Don't use deprecated np.asscalar() (#2800) It got deprecated in numpy 1.16 and throws a ton of warnings due to that. All the function does is returning .item() anyway, which is why it got deprecated. * Add support for cftime.datetime coordinates with coarsen (#2778) * some docs updates (#2746) * Friendlier io title. * Fix lists. * Fix *args, **kwargs "inline emphasis..." * misc * Reference xarray_extras for csv writing. Closes #2289 * Add metpy accessor. Closes #461 * fix transpose docstring. Closes #2576 * Revert "Fix lists." This reverts commit 39983a5. * Revert "Fix *args, **kwargs" This reverts commit 1b9da35. * Add MetPy to related projects. * Add Weather and Climate specific page. * Add hvplot. * Note open_dataset, mfdataset open files as read-only (closes #2345). * Update metpy 1 Co-Authored-By: dcherian <dcherian@users.noreply.github.com> * Update doc/weather-climate.rst Co-Authored-By: dcherian <dcherian@users.noreply.github.com> * Drop failing tests writing multi-dimensional arrays as attributes (#2810) These aren't valid for netCDF files. Fixes GH2803 * Push back finalizing deprecations for 0.12 (#2809) 0.12 will already have a big change in dropping Python 2.7 support. I'd rather wait a bit longer to finalize these deprecations to minimize the impact on users. * enable loading remote hdf5 files (#2782) * attempt at loading remote hdf5 * added a couple tests * rewind bytes after reading header * addressed comments for tests and error message * fixed pep8 formatting * created _get_engine_from_magic_number function, new tests * added description in whats-new * fixed test failure on windows * same error on windows and nix * Release 0.12.0 * Add whats-new for 0.12.1 * Rework whats-new for 0.12 * DOC: Update donation links * DOC: remove outdated warning (#2818) * Allow expand_dims() method to support inserting/broadcasting dimensions with size>1 (#2757) * Make using dim_kwargs for python 3.5 illegal -- a ValueError is thrown * dataset.expand_dims() method take dict like object where values represent length of dimensions or coordinates of dimesnsions * dataarray.expand_dims() method take dict like object where values represent length of dimensions or coordinates of dimesnsions * Add alternative option to passing a dict to the dim argument, which is now an optional kwarg, passing in each new dimension as its own kwarg * Add expand_dims enhancement from issue 2710 to whats-new.rst * Fix test_dataarray.TestDataArray.test_expand_dims_with_greater_dim_size tests to pass in python 3.5 using ordered dicts instead of regular dicts. This was needed because python 3.5 and earlier did not maintain insertion order for dicts * Restrict core logic to use 'dim' as a dict--it will be converted into a dict on entry if it is a str or a sequence of str * Don't cast dim values (coords) as a list since IndexVariable/Variable will internally convert it into a numpy.ndarray. So just use IndexVariable((k,), v) * TypeErrors should be raised for invalid input types, rather than ValueErrors. * Force 'dim' to be OrderedDict for python 3.5 * Allow expand_dims() method to support inserting/broadcasting dimensions with size>1 (#2757) * use .size attribute to determine the size of a dimension, rather than converting to a list, which can be slow for large iterables * Make using dim_kwargs for python 3.5 illegal -- a ValueError is thrown * dataset.expand_dims() method take dict like object where values represent length of dimensions or coordinates of dimesnsions * dataarray.expand_dims() method take dict like object where values represent length of dimensions or coordinates of dimesnsions * Add alternative option to passing a dict to the dim argument, which is now an optional kwarg, passing in each new dimension as its own kwarg * Add expand_dims enhancement from issue 2710 to whats-new.rst * Fix test_dataarray.TestDataArray.test_expand_dims_with_greater_dim_size tests to pass in python 3.5 using ordered dicts instead of regular dicts. This was needed because python 3.5 and earlier did not maintain insertion order for dicts * Restrict core logic to use 'dim' as a dict--it will be converted into a dict on entry if it is a str or a sequence of str * Don't cast dim values (coords) as a list since IndexVariable/Variable will internally convert it into a numpy.ndarray. So just use IndexVariable((k,), v) * TypeErrors should be raised for invalid input types, rather than ValueErrors. * Force 'dim' to be OrderedDict for python 3.5 * Allow expand_dims() method to support inserting/broadcasting dimensions with size>1 (#2757) * Move enhancement description up to 0.12.1 * use .size attribute to determine the size of a dimension, rather than converting to a list, which can be slow for large iterables * Make using dim_kwargs for python 3.5 illegal -- a ValueError is thrown * dataset.expand_dims() method take dict like object where values represent length of dimensions or coordinates of dimesnsions * dataarray.expand_dims() method take dict like object where values represent length of dimensions or coordinates of dimesnsions * Add alternative option to passing a dict to the dim argument, which is now an optional kwarg, passing in each new dimension as its own kwarg * Add expand_dims enhancement from issue 2710 to whats-new.rst * Fix test_dataarray.TestDataArray.test_expand_dims_with_greater_dim_size tests to pass in python 3.5 using ordered dicts instead of regular dicts. This was needed because python 3.5 and earlier did not maintain insertion order for dicts * Restrict core logic to use 'dim' as a dict--it will be converted into a dict on entry if it is a str or a sequence of str * Don't cast dim values (coords) as a list since IndexVariable/Variable will internally convert it into a numpy.ndarray. So just use IndexVariable((k,), v) * TypeErrors should be raised for invalid input types, rather than ValueErrors. * Force 'dim' to be OrderedDict for python 3.5

naught101 · 2019-04-12T07:24:43Z

Is there currently a way to drop unused coordinates, like this?

naught101 · 2019-04-12T07:26:07Z

I guess ds = ds.drop([c for c in ds.coords if c not in ds.dims]) works. Might be nice to have a convenience function for that though.

kmsquire force-pushed the kms/drop_dims branch from 14853f0 to cf73e3c Compare February 13, 2019 04:37

kmsquire mentioned this pull request Feb 13, 2019

Removing dimensions from Dataset objects #1949

Closed

kmsquire force-pushed the kms/drop_dims branch from cf73e3c to 6a0f04b Compare February 14, 2019 19:31

kmsquire force-pushed the kms/drop_dims branch from fb847e0 to 5532cbb Compare February 22, 2019 16:46

shoyer reviewed Feb 24, 2019

View reviewed changes

xarray/core/dataset.py Outdated Show resolved Hide resolved

xarray/core/dataset.py Outdated Show resolved Hide resolved

doc/indexing.rst Outdated Show resolved Hide resolved

doc/indexing.rst Outdated Show resolved Hide resolved

kmsquire force-pushed the kms/drop_dims branch from 5532cbb to 539e65e Compare February 26, 2019 19:01

kmsquire force-pushed the kms/drop_dims branch from 539e65e to c08fb5f Compare February 26, 2019 19:03

shoyer reviewed Feb 27, 2019

View reviewed changes

kmsquire force-pushed the kms/drop_dims branch from c08fb5f to e7fa3b3 Compare March 1, 2019 06:35

kmsquire added 2 commits March 2, 2019 18:27

ENH: Add Dataset.drop_dims()

3f4552d

* Drops full dimensions and any corresponding variables in a Dataset * Fixes GH1949

DOC: Add Dataset.drop_dims() documentation

2cc10bd

kmsquire force-pushed the kms/drop_dims branch from c16938c to 2cc10bd Compare March 3, 2019 02:27

shoyer merged commit e8eb83b into pydata:master Mar 3, 2019

pletchm pushed a commit to pletchm/xarray that referenced this pull request Mar 21, 2019

Add Dataset.drop_dims (pydata#2767)

93185ef

* ENH: Add Dataset.drop_dims() * Drops full dimensions and any corresponding variables in a Dataset * Fixes GH1949 * DOC: Add Dataset.drop_dims() documentation

pletchm pushed a commit to pletchm/xarray that referenced this pull request Mar 21, 2019

Add Dataset.drop_dims (pydata#2767)

ab268de

* ENH: Add Dataset.drop_dims() * Drops full dimensions and any corresponding variables in a Dataset * Fixes GH1949 * DOC: Add Dataset.drop_dims() documentation

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `Dataset.drop_dims` #2767

Add `Dataset.drop_dims` #2767

kmsquire commented Feb 13, 2019 •

edited

Loading

pep8speaks commented Feb 13, 2019 •

edited

Loading

kmsquire commented Feb 13, 2019

kmsquire commented Feb 21, 2019

shoyer commented Feb 22, 2019

kmsquire commented Feb 22, 2019

kmsquire commented Feb 26, 2019

shoyer Feb 27, 2019

kmsquire Mar 1, 2019

kmsquire commented Mar 3, 2019

shoyer commented Mar 3, 2019

kmsquire commented Mar 4, 2019

naught101 commented Apr 12, 2019

naught101 commented Apr 12, 2019

Add Dataset.drop_dims #2767

Add Dataset.drop_dims #2767

Conversation

kmsquire commented Feb 13, 2019 • edited Loading

pep8speaks commented Feb 13, 2019 • edited Loading

Comment last updated at 2019-03-03 02:27:43 UTC

kmsquire commented Feb 13, 2019

kmsquire commented Feb 21, 2019

shoyer commented Feb 22, 2019

kmsquire commented Feb 22, 2019

kmsquire commented Feb 26, 2019

shoyer Feb 27, 2019

Choose a reason for hiding this comment

kmsquire Mar 1, 2019

Choose a reason for hiding this comment

kmsquire commented Mar 3, 2019

shoyer commented Mar 3, 2019

kmsquire commented Mar 4, 2019

naught101 commented Apr 12, 2019

naught101 commented Apr 12, 2019

Add `Dataset.drop_dims` #2767

Add `Dataset.drop_dims` #2767

kmsquire commented Feb 13, 2019 •

edited

Loading

pep8speaks commented Feb 13, 2019 •

edited

Loading